Add PyCapsule Protocol support for Arrow inputs#1265
Open
H0TB0X420 wants to merge 3 commits intoapache:mainfrom
Open
Add PyCapsule Protocol support for Arrow inputs#1265H0TB0X420 wants to merge 3 commits intoapache:mainfrom
H0TB0X420 wants to merge 3 commits intoapache:mainfrom
Conversation
- Add Protocol types for Arrow PyCapsule Interface - Update schema parameters to accept any Arrow-compatible library - Move pyarrow to TYPE_CHECKING (optional at runtime)
kylebarron
reviewed
Oct 7, 2025
Member
|
I'm not sure we can remove the dependency unless we do something about our auto-literal conversion. I suspect there are other places. https://github.com/apache/datafusion-python/blob/main/python/datafusion/expr.py#L563 |
Member
|
Perhaps we should add support for the protocol without removing the pyarrow dependency yet |
Contributor
Author
|
I've updated the PR based on your feedback. PyArrow will stay required for now. That way people can at least use nanoarrow/Polars schemas, and we can tackle making PyArrow fully optional down the road. All tests passing! Let me know if you'd like any other changes. I am also happy to break this into smaller steps and handle the full optional PyArrow work in follow up issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1227
Rationale for this change
(From original issue)
PyArrow is a massive dependency (>100MB unpacked) and the only required dependency for datafusion-python. Many Python Arrow libraries implement the PyCapsule Interface, allowing users to choose lightweight alternatives like nanoarrow (~7MB), arro3, or pass data directly from Polars, DuckDB, etc.
This PR implements the first phase of making PyArrow optional by updating input parameters to accept any Arrow-compatible library via the PyCapsule Interface.
What changes are included in this PR?
ArrowSchemaExportableregister_csv,register_parquet,register_json,register_avro,register_listing_table, and read methods to acceptArrowSchemaExportableTYPE_CHECKINGblock (optional at runtime for type hints only)Note: This PR covers input parameters only. Return types (ToPyArrow conversions) still reference pyarrow and will be addressed in a follow-up PR.
Are there any user-facing changes?
Breaking changes: None. All existing PyArrow usage continues to work.
New functionality: Users can now pass Arrow schemas from any library implementing
__arrow_c_schema__()(nanoarrow, arro3, Polars, DuckDB, etc.) to datafusion methods.Type hints: Schema parameters now show
ArrowSchemaExportable | Noneinstead ofpa.Schema | None, but accept both.